fix(cluster_setup): stop reverting artifacts.yaml on repeat installs,… - #134
Conversation
Covers connecting an external Splunk Enterprise instance to the SAIA backend — JWT signing key setup, issuer_uri fix, mixed-content options (disable SSL workaround + generic TLS termination via LB/ingress), AIPlatform CR patching to avoid operator revert, and correct Splunk restart procedure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… runbook - Step 2: verify port 8089 reachability from k0s cluster nodes, not laptop - Step 3 Option B: fix saia_sok_url stanza to [saia_sok_configurations] - Step 4: replace incorrect AIPlatform CR patch with direct ConfigMap edit; explain that splunkConfiguration.endpoint is the HEC endpoint (not the JWT issuer), and that the operator only fills missing ConfigMap keys so a direct SPLUNK_ISSUERS edit is safe and persistent Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…unk runbook - fix(P3): broken Step 3 table anchor pointed at old Traefik heading; update to match actual Option B heading (TLS Termination via LB/Ingress) - fix(P3): curl HTTPS check after disabling Splunk Web SSL expected "connection refused" but port 8000 stays open speaking HTTP — a TLS client gets a handshake/protocol error, not ECONNREFUSED; updated expected output to match TLS negotiation failure - fix(P3): restart-as-owner instructions only covered the case where the SSH user IS the owner; added sudo -H -u <owner> form for the common case where admins log in as ec2-user but Splunk runs as 'splunk' - fix(P2): Cleanup section still referenced an AIPlatform CR patch that was already corrected in Step 4; replaced with direct ConfigMap patch for the both-issuers case; also replaced stale AIService endpoint check with a direct SPLUNK_ISSUERS ConfigMap verification Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nk runbook - fix(P2): architecture diagram showed Browser→Splunk→SAIA implying Splunk proxies API calls; browser calls SAIA directly with the JWT token Splunk issued. Diagram and note now show the correct two-path flow: browser→Splunk for auth, browser→SAIA for all AI Assistant API calls. Firewall note added (SAIA must be reachable from browser network, not just from Splunk host) - fix(P3): overview said allowlist is controlled via AIPlatform CR→AIService; the reconciler seeds SPLUNK_ISSUERS directly in the ConfigMap. Replaced the stale propagation chain with "ConfigMap <name>-saia-config" reference matching Step 4 - fix(P1): Step 4 patch examples used <PUBLIC_IP> — SPLUNK_ISSUERS must be the exact issuer_uri value (IP or FQDN) from authentication.conf; a mismatch causes persistent 401/issuer-not-allowed even after patching. Replaced with <EXACT_ISSUER_URI> and added an explanatory note Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…doc-v2 docs(cluster_setup): add external Splunk integration runbook
…o to v0.52.0 Addresses VULN-96668 (golang.org/x/net) and VULN-96647 (golang.org/x/crypto). Also pulls in transitive upgrades: sys v0.45.0, term v0.43.0, text v0.37.0, tools v0.44.0. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… warn on mutable image tags configure_images() snapshotted artifacts.yaml/splunk-operator-cluster.yaml to a .original file on first run and restored from it on every subsequent run before re-injecting image values. Since .original was never refreshed, any legitimate change to those manifests (new operator release, new env var, new sidecar) was silently reverted on every re-install after the first. Drop the snapshot/restore step; the sed substitutions already only touch their own named fields, so rewriting in place is idempotent. Also warn in validate_image_config() when a configured image has no tag or uses a mutable tag (:latest, :preview, :stable*, :dev, :nightly), since imagePullPolicy: IfNotPresent means re-running install without a new distinct tag will not actually upgrade the running image.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b8c88dd52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The image: substitutions for the AI operator and Splunk operator matched by content (*splunk*ai*operator*, *splunk*operator*) against the image string itself. This only matched the manifest's pristine default image; once a custom image (e.g. a private registry with no such substring) was written in on a prior run, a later run with a new tag would silently fail to match, leaving the stale image in place. This was previously masked by a since-removed snapshot/restore step that reset the manifest to pristine before each run. Anchor both substitutions on the unique env entry that always immediately precedes the target image: line (RAY_VERSION for artifacts.yaml, POD_NAME for splunk-operator-cluster.yaml) instead of matching image content. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b07745c4ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…rades fix(deps): upgrade golang.org/x/net to v0.55.0 and golang.org/x/crypt…
Three follow-up issues from PR review on the prior idempotency fix: - Restore the tools/cluster_setup/*.original .gitignore rule (narrowed comment): eks_cluster_with_stack.sh still snapshots/restores .original backups, so deleting the rule left those EKS byproducts untracked in every checkout even though k0s_cluster_with_stack.sh no longer needs it. - Parse the tag from the last path segment only in the mutable-tag check, not from the whole image string. A registry port (e.g. localhost:5000/team/saia-api) was previously read as the tag, so untagged images behind a ported registry silently skipped the no-tag warning. - Extend the mutable-tag check to cover images.fluentBit.image and images.nginx.image (always patched via RELATED_IMAGE_FLUENT_BIT/ RELATED_IMAGE_NGINX) and images.splunk.image (patched via RELATED_IMAGE_SPLUNK_ENTERPRISE, guarded by SPLUNK_MODE=internal since it's only deployed in that mode) — these were patched into the manifest but never checked for the same same-tag-rerun hazard. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75e75ee8b3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
configure_images() patches RELATED_IMAGE_OTEL_COLLECTOR into the manifest, but images.otelCollector.image was missing from the mutable-tag validation list, so an untagged or mutable-tagged OTel collector image silently skipped the same-tag-rerun warning. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c271c4ea1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
configure_images() patches images.splunk.operatorImage into splunk-operator-cluster.yaml (via the POD_NAME-anchored image: sed) in internal mode, but the mutable-tag validation loop never checked SPLUNK_OPERATOR_IMAGE — so a tagless or mutable-tagged operator image was validated without warning. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add two unit-test suites to test_k0s_cluster_with_stack.sh: - configure_images upgrade idempotency: reproduces the exact regression fixed in b07745c — a custom private-registry image containing none of "splunk"/"ai"/"operator" must be correctly overwritten on a second ("upgrade") run, not left stale. - validate_image_config mutable-tag warnings: covers the registry-port tag-parsing fix and every RELATED_IMAGE_* field configure_images patches (operator, ray, weaviate, saia x3, fluentBit, nginx, otelCollector, splunk + splunk-operator under internal mode), so any future field added to one list without the other gets caught. Track test_installer_dry_run.sh (previously untracked) and add a new cluster-setup-tests.yml workflow that runs both bash test suites, wired into main.yml alongside unit-tests. Neither suite needs a cluster, kubectl, or network access. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… warn on mutable image tags
configure_images() snapshotted artifacts.yaml/splunk-operator-cluster.yaml to a .original file on first run and restored from it on every subsequent run before re-injecting image values. Since .original was never refreshed, any legitimate change to those manifests (new operator release, new env var, new sidecar) was silently reverted on every re-install after the first. Drop the snapshot/restore step; the sed substitutions already only touch their own named fields, so rewriting in place is idempotent.
Also warn in validate_image_config() when a configured image has no tag or uses a mutable tag (:latest, :preview, :stable*, :dev, :nightly), since imagePullPolicy: IfNotPresent means re-running install without a new distinct tag will not actually upgrade the running image.
Description
Related Issues
Type of Change
Changes Made
Testing Performed
make test)make lint)Test Environment
Test Steps
Documentation
Checklist
Breaking Changes
Impact:
Migration Path:
Screenshots/Recordings
Additional Notes
Reviewer Notes
Please pay special attention to:
Commit Message Convention: This PR follows Conventional Commits